Recognition of Transitive Actions with Hierarchical Neural Network Learning

The recognition of actions that involve the use of objects has remained a challenging task. In this paper, we present a hierarchical selforganizing neural architecture for learning to recognize transitive actions from RGB-D videos. We process separately body poses extracted from depth map sequences and object features from RGB images. These cues are subsequently integrated to learn action–object mappings in a selforganized manner in order to overcome the visual ambiguities introduced by the processing of body postures alone. Experimental results on a dataset of daily actions show that the integration of action–object pairs significantly increases classification performance.